Skip to content

golive: journal the measured handover delay; load-proof gap timing test (#534) - #554

Merged
scuttletoby merged 1 commit into
mainfrom
fix/golive-534-handover-delay
Nov 20, 2026
Merged

scuttletoby merged 1 commit into
mainfrom
fix/golive-534-handover-delay

Conversation

@scuttletoby

Copy link
Copy Markdown
Owner

Fixes #534

What

  • The hold. There is no fixed hold left to lower. fix(golive): hand the home over only to the service's own launcher #548 replaced "held on 3 polls / 100 ms" with a check that the boot service's main process holds the serving lock (/proc/locks). So the kit already lets go at its first look after the launcher's claim. Looks run every 20 ms, well under a 60 ms target and well inside the server's 250 ms busy_timeout. An SSH probe is a separate process, so it can never pass that check, whatever the timing.
  • Logging the delay. Gap.starting() measures how long the owner lock was kept after the claim. The count runs from the start of the last look that did not find the launcher to the let-go, so it is an upper bound. The verify step journals it as handoverMs. When it reaches the server's 250 ms wait, the step also writes handoverLate, which names a server that may have idled on a very busy host.
  • The flaky bound. gap.test.py used assertLess(free, 0.25) in wall-clock time, which a slow runner can break. It now checks the order of the kit's looks instead: every look that began after the claim found the launcher, and there was exactly one. The wait for the free lock is a condition with the suite's 300 s cap, not a fixed number of sleeps.

State question (tests)

  • Runs again: a second starting() on the same gap resets handover_ms. A later timeout carries no old delay.
  • Stopped half-way: a start that raises journals start failed, no handoverMs, and the owner lock is free.
  • Missing state: with no gap open, served() returns {'marker': None}. A timeout handover has no handoverMs.
  • Lateness: whether a handover counts as late is fixed by patching the wait, never by how fast the runner is.

General vs project

Everything here is general to the kit: upstream's 250 ms owner-lock wait (pinned in contract.test.py) and its serving lock. Nothing is specific to one machine or project.

Old code

Test On main's handoff.py
test_the_handover_delay_is_measured_and_journalled fails (SERVER_LOCK_WAIT_MS/handoverMs missing)
test_a_probe_loop_on_the_serving_lock_is_not_the_launcher fails (handover_ms missing); its look-order check passes, as the old let-go was already prompt. Only the timing bound was flaky

I ran both against main's handoff.py. All golive suites pass locally: back, check, contract, copy, gap, journal, rehearse, switch.

🤖 Generated with Claude Code

…st (#534)

The kit lets go of the owner lock at its first look after the launcher's
claim of the serving lock (#548 already replaced the 100 ms hold with the
main-process check). The verify step now journals handoverMs (an upper
bound on claim -> let-go) and handoverLate when it reached the 250 ms the
server waits for the owner lock.

The probe-loop test no longer bounds wall-clock time: it checks that the
kit let go at its first look that began after the claim, which a slow
runner cannot break.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

Review (lens A)

No blocking findings.

Checked: handoff.py watcher change, served() facts, the two new/changed gap.test.py tests, README. gap.test.py passes on the branch (29 tests). With main's handoff.py both new tests error, as the PR body says. Note that the look-order check passes on the old code too: only the timing bound was flaky.

On the brief's question: does the let-go still beat an SSH probe and the server's 250 ms wait under load? Yes. A probe can never pass, because the check is "the service's main pid holds the serving lock" in /proc/locks, not a hold time. One look costs one /proc/locks read plus one systemctl --user show (about 5 ms here at load 3 on 12 cores), and looks run every 20 ms. So the let-go comes about 25 ms after the claim in normal conditions. The server's 250 ms wait only starts after the child has spawned and Node has booted, so the real margin is larger than 250 ms minus the let-go time. Upstream's cost of missing it is real, though: #idleWhileStateDirOwned marks a pending trial failed: state-dir-owned and polls every 30 s. So the journal fact is worth having.

MINOR

  1. tests/gap.test.py:721-722: the new look-order assertion can still flake. claimed is stamped after BEGIN EXCLUSIVE returns. If the watcher thread gets the GIL in between, it records a look whose began is before claimed[0] with found=True, and returns. Then [found for at, found in looks if at >= claimed[0]] is [], not [True]. The window is small but load makes it bigger, which is the same failure mode t3-golive: owner-lock handover margin shrank to ~110 ms (#531 verify nit) #534 set out to remove. Fix: stamp claimed just before BEGIN EXCLUSIVE. The spy only records looks that found the lock held, and the probes have stopped by then, so the list after that stamp is still exactly [True].
  2. handoff.py:263: handoverMs/handoverLate are journalled only by served(), so only on a verify that passed. The case the fact exists for is a late handover where the server idled. There, verify often fails (or the switch puts T3 back), served() is never called, and the journal keeps no delay. Fix: also journal handover/handoverMs in the update step's facts right after starting() returns (W1/W2/W3 _update, switch _update, put-back).
  3. handoff.py:211: each look that finds the lock held calls systemctl show. While an SSH launch probes in a loop, this runs on almost every look, and it is the slowest part of a look on a loaded host. Optional: compare the holders with the last-read mainPid first, and re-read the service only on a miss.
  4. Nit: looked is set when the watcher's body first runs, not before watcher.start(). If that thread is scheduled late, the "upper bound" could come in under the real delay. Set it in starting() before watcher.start().

State question: a rerun (the second starting() resets handover_ms), a failed start (no handoverMs, lock free), and no gap ({'marker': None}) are all covered by the new test. No persisted state is added, so an older kit's journal reads the same.

@scuttletoby

Copy link
Copy Markdown
Owner Author

Review (lens B)

Head 24c105f. Read #534, the diff and the body; ran all golive suites on this head (all pass).
Mutation check: I put a fixed hold back in the watcher (let go only on the 2nd look that finds the launcher). test_a_probe_loop_on_the_serving_lock_is_not_the_launcher then fails ([True, True] != [True]), and so does test_the_handover_delay_is_measured_and_journalled ('timeout' != 'launcher' with the 0.05 s fixture wait). So the new look-order assertion really guards the "no fixed hold" rule, with no wall-clock bound.

No blocking findings.

Brief question, "does the hold still beat an SSH probe and the server's 250 ms under load?":

  • The probe: yes, and whatever the timing. The check is by identity (the service's main pid among the holders in /proc/locks), so a probe from another process never passes.
  • The 250 ms: the kit's delay is one 20 ms poll, plus a read of /proc/locks, plus one systemctl --user show (fork, exec and a D-Bus round trip), plus the close. The server's own 250 ms only starts after the launcher has spawned its child and Node has booted, so the real margin is larger than handoverMs suggests. A heavily swapping host can still stretch systemctl past it, and that is now journalled (handoverLate).

MINOR

  1. systemctl sits on the critical path of every look. lib/golive/handoff.py:209-217. Every look that sees the serving lock held calls service() (systemctl --user show) before it can let go, so the slowest part of the handover is a subprocess. Suggestion: read service() on the looks where the serving lock is not held, or once fn returned and the unit is active, and keep the last mainPid. Call it again only when holds_lock(cached_pid) is false. Then a launcher's claim costs only a /proc/locks read before the let-go.
  2. handoverLate can name a server that did not idle. lib/golive/handoff.py:265. It compares an upper bound counted from the claim with the 250 ms that starts later, at the server's own BEGIN EXCLUSIVE, so it can name a server that did not idle. The wording "may have idled" already allows for that. Fine as a signal; just do not read it as proof.

State question: a rerun resets handover_ms; a raising start leaves no handoverMs and frees the lock; with no gap open, served() returns {'marker': None}. All covered by tests. Merges cleanly under #556 and with #557.

Verdict: SAFE.

@scuttletoby
scuttletoby merged commit f49745f into main Nov 20, 2026
27 checks passed
scuttletoby added a commit that referenced this pull request Nov 20, 2026
…st (#534) (#554)

The kit lets go of the owner lock at its first look after the launcher's
claim of the serving lock (#548 already replaced the 100 ms hold with the
main-process check). The verify step now journals handoverMs (an upper
bound on claim -> let-go) and handoverLate when it reached the 250 ms the
server waits for the owner lock.

The probe-loop test no longer bounds wall-clock time: it checks that the
kit let go at its first look that began after the claim, which a slow
runner cannot break.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
scuttletoby added a commit that referenced this pull request Nov 20, 2026
…cks is blind (#530)

Lens round on #556:
- The handover keeps the holder for every outcome but the launcher's claim while the service is not active: a
  `t3 update` that exits non-zero or times out, and a target that never comes up, no longer free the home.
- hold_home starts no holder when /proc/locks cannot be read (it could never be found, adopted or stopped); the
  run holds the home itself and journals why. status and check name an active holder unit they cannot see there.
- A reboot ending the holder is a known gap, documented; status now says the home is free then.
- The switch guard's fail-open start leaves a disabled T3 (its W1 holds it) stopped and held.
- holder_said: do not start T3 by hand while it holds.
- #554's look-order test stamps the claim before BEGIN EXCLUSIVE (no GIL race); `looked` is set before the
  watcher starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scuttletoby added a commit that referenced this pull request Nov 20, 2026
…cks is blind (#530)

Lens round on #556:
- The handover keeps the holder for every outcome but the launcher's claim while the service is not active: a
  `t3 update` that exits non-zero or times out, and a target that never comes up, no longer free the home.
- hold_home starts no holder when /proc/locks cannot be read (it could never be found, adopted or stopped); the
  run holds the home itself and journals why. status and check name an active holder unit they cannot see there.
- A reboot ending the holder is a known gap, documented; status now says the home is free then.
- The switch guard's fail-open start leaves a disabled T3 (its W1 holds it) stopped and held.
- holder_said: do not start T3 by hand while it holds.
- #554's look-order test stamps the claim before BEGIN EXCLUSIVE (no GIL race); `looked` is set before the
  watcher starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scuttletoby added a commit that referenced this pull request Nov 20, 2026
…530) (#556)

* golive: keep the home owned after a W1 or switch that stops T3 ends (#530)

A W1 (and a switch) may fail or be killed with T3 stopped on purpose (the home may be half-restored). The owner
lock it took after the stop was held by the run's own process, so it was gone with the run: the next SSH launch
of the desktop app found the home free and started its own server on it.

Right after the stop the kit now starts a holder, a transient unit of its own (t3-golive-hold-<hash of the home>,
running `t3-golive hold --home <home>`), that takes the owner lock and keeps it until stopped; the run adopts it
as its lock. Every start the kit makes goes through the handover, so the new launcher gets the home from the
holder. A failure before the start, a failed start or a kill leaves the holder holding. The same back's rerun
takes it over (no free moment, one unit name per home); a resumed verify lets a leftover holder go once the
service runs T3; status and check name it; settle stops it; a reboot ends it. It is found by its live process in
/proc/locks, never by a file. Where no transient unit starts, the run holds the lock itself, as before, and says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* golive: format the README table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* golive: keep the holder until T3 really runs; no holder when /proc/locks is blind (#530)

Lens round on #556:
- The handover keeps the holder for every outcome but the launcher's claim while the service is not active: a
  `t3 update` that exits non-zero or times out, and a target that never comes up, no longer free the home.
- hold_home starts no holder when /proc/locks cannot be read (it could never be found, adopted or stopped); the
  run holds the home itself and journals why. status and check name an active holder unit they cannot see there.
- A reboot ending the holder is a known gap, documented; status now says the home is free then.
- The switch guard's fail-open start leaves a disabled T3 (its W1 holds it) stopped and held.
- holder_said: do not start T3 by hand while it holds.
- #554's look-order test stamps the claim before BEGIN EXCLUSIVE (no GIL race); `looked` is set before the
  watcher starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

t3-golive: owner-lock handover margin shrank to ~110 ms (#531 verify nit)

1 participant